Multi-modal Masked Autoencoders for Medical Vision-and-Language Pre-training

Medical vision-and-language pre-training provides a feasible solution to extract effective representations from medical images and texts. However, few studies have been dedicated this field facilitate understanding. In paper, we propose self-supervised learning paradigm with multi-modal masked autoencoders (M $$^3$$ AE), which learn cross-modal domain knowledge by reconstructing missing pixels tokens randomly There are three key designs make simple approach work. First, considering the different information densities of vision language, adopt masking ratios for input image text, where considerably larger ratio is used images. Second, use visual textual features layers perform reconstruction deal levels abstraction in language. Third, develop language decoders (i.e., Transformer multi-layer perceptron language). To comprehensive evaluation further research, construct benchmark including tasks. Experimental results demonstrate effectiveness our approach, state-of-the-art achieved on all downstream Besides, conduct analysis better verify components various settings pre-training. The source code available at https://github.com/zhjohnchan/M3AE .